Hands-free augmented reality system for picking and/or sorting assets

ABSTRACT

A system is provided for hands-free handling of at least one asset by a user. The system includes a user device configured to be worn by a user and a control system remotely located relative to the user device and configured to exchange asset-related data with the user device, the asset-related data including at least one or more notifications related to the handling of the at least one asset by the user at the asset location. The user device contains a processor configured to determine location data associated with the at least one asset and including a location for the at least one asset, the determination being based, in part, upon the obtained asset identifier data; dynamically generate and display, at the user device, one or more navigational projections configured to guide the user to the asset location; and detect handling of the at least one asset by the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/103,566, filed Aug. 14, 2018, which claims priority to U.S.Provisional Patent Application No. 62/545,752 entitled “HANDS-FREEAUGMENTED REALITY SYSTEM FOR PICKING AND/OR SORTING ASSETS AND METHODSOF UTILIZING THE SAME,” filed Aug. 15, 2017, each of which is hereinincorporated by reference in its entirety.

BACKGROUND

The automated handling of parcels (e.g., packages, containers, letters,items, pallets, etc.) transported by common carriers throughtransportation networks is a complex problem with many parts. No singlesystem or method alone appears to provide a comprehensive solution forall conditions. A primary component in some systems and methods forautomated handling of packages is a conveyance device (i.e., a conveyorbelt), which is generally formed and/or extended around at least twodriving wheels. Thus, by turning the driving wheels, the conveyor beltmay run continuously. Conveyor belts may also generally be flexible anddeformable at least while running in contact with the driving wheels,and a multitude of materials, linkages, and so forth have been used toachieve these goals.

Where automated handling of packages has been implemented, certaininefficiencies may arise. For example, where packages may be improperlyor too closely placed relative to one another on a conveyor belt,congestion may arise, impacting various measurements or the like thatneed to be performed on the packages while on the conveyor belt. Stillfurther, where the materials in which packages are wrapped (e.g., foilor paper or the like) differ in color or other material characteristics,inaccuracies may also arise in any measurements, imaging, orobservations made in an automated fashion relative to the packages orassets.

Beyond interactions with conveyor belts, automated handling of parcelscreates additional challenges related to how the parcels—referred toelsewhere herein as assets—are transported and/or handled by carrierpersonnel between conveyor belts (and the like) and respective inventorylocations (e.g., for picking) and/or sort locations (e.g., for sorting)associated with the assets.

In this context, a need exists for improved technological systems,assemblies, and/or methods for maintaining accurate records of thelocation of an asset in a sort and/or pick process, while also providingto carrier personnel improved instructions and/or guidance for theautomated handling of the packages within various environments (e.g., awarehouse environment whether relative to a sort location, a picklocation, a conveyor belt, and/or any combination thereof).

BRIEF SUMMARY

According to various embodiments described herein, a system is providedfor hands-free handling of at least one asset by a user. The system caninclude a user device configured to be worn by a user. The user devicemay include one or more memories and one or more processors configuredto perform the following operations. Asset identifier data can beobtained for at least one asset. Location data, associated with alocation for the at least one asset, can be determined based, at leastin part, upon the obtained asset identifier data. One or morenavigational projections configured to guide the user to the locationcan be dynamically generated and displayed. Handling of the at least oneasset by the user can be detected. One or more notifications associatedwith the handling of the at least one asset by the user at the locationmay be received.

In another embodiment, a computer implemented method is provided forhands-free handling of at least one asset by a user is provided. Themethod may include the following operations. Asset identifier data forat least one asset can be received from a remote location relative to auser device that is worn by the user. First location data associatedwith the user device can be determined at the user device. Secondlocation data associated with the at least one asset may be determined.The second location data may be determined, based at least in part, onanalyzing the received asset identifier data. The first location datamay be determined, based at least in part, on analyzing a presentposition of the user device. One or more navigational projectionsconfigured to guide the user to a location associated with the secondlocation data may be dynamically generated and displayed. The one ormore navigational projections may be dynamically updated based at leastin part on one or more detected changes of a present location of theuser device.

In yet another embodiment, a computer program product is provided forhands-free handling of at least one asset by a user. The computerprogram product may include at least one non-transitorycomputer-readable storage medium having computer-readable program codeportions stored therein. The computer-readable program code portions mayinclude one or more executable portions configured for performing thefollowing operations. An environment that a user is located in can bemapped based at least in part on generating a multidimensional graphicalrepresentation of the environment. Asset identifier data to identify atleast one asset can be received at a user device. One or more assetslocations can be associated within the mapped environment. The one ormore asset locations may be associated with the at least one asset. Oneor more navigational projections configured to guide the user to anasset location within the environment may be generated and displayedbased at least on the associating and within the environment that theuser is in.

Additional details are provided in the accompanying Appendix, whichinclude a visual presentation regarding various features of theembodiments described herein.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

FIG. 1 schematically depicts a control system according to one or moreembodiments shown and described herein;

FIG. 2 schematically depicts a control system according to one or moreembodiments shown and described herein;

FIG. 3 schematically depicts a user device that communicates with acontrol system of according to one or more embodiments shown anddescribed herein;

FIG. 4 depicts a user device in conjunction with a harness mechanismaccording to one or more embodiments shown and described herein;

FIG. 5 depicts a user device in isolation without the harness mechanismaccording to one or more embodiments shown and described herein;

FIG. 6 schematically depicts a flowchart illustrating operations andprocesses performed by a user device according to one or moreembodiments shown and described herein;

FIG. 7 schematically depicts a flowchart illustrating operations andprocesses performed according to one or more embodiments shown anddescribed herein;

FIG. 8 depicts a facility and an environmental mapping procedureachieved via a user device according to one or more embodiments shownand described herein;

FIG. 9 depicts a facility and a pathway indicating navigationalprojection achieved via a user device according to one or moreembodiments shown and described herein;

FIG. 10 depicts a shelving containing portion of a facility and aplacement indicating navigational projection achieved via a user deviceaccording to one or more embodiments shown and described herein;

FIGS. 11A-C depict further views of three exemplary embodiments of theplacement indicating navigational projection achieved via a user deviceaccording to one or more embodiments shown and described herein;

FIG. 12 is a perspective or isometric view of a conveyor belt assemblythat may be utilized in conjunction with the control system and userdevice according to one or more embodiments shown and described herein;

FIGS. 13A-13F depict further views of additional exemplary navigationalprojections achieved via a user device and in conjunction with aconveyor belt assembly according to one or more embodiments shown anddescribed herein;

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

The present disclosure will now be described more fully hereinafter withreference to the accompanying drawings, in which some, but not allembodiments of the disclosure are shown. Indeed, the disclosure may beembodied in many different forms and should not be construed as limitedto the embodiments set forth herein. Rather, these embodiments areprovided so that this disclosure will satisfy applicable legalrequirements. Like numbers refer to like elements throughout.

I. Overview

Existing asset handling technologies remain burdensome and are notsuitable for all tasks. For example, augmented reality-based computingsolutions have been pursued, such as with reference to U.S. Ser. No.15/581,609, the contents of which as are incorporated by referenceherein in their entirety. These augmented-reality-based solutions canutilize objects, such as smart glasses, to generate an environment so asto provide to carrier personnel (e.g., via a lens of the smart glasses)directions for transporting particular assets or packages. However,smart glasses may be uncomfortable to use for long periods of time(e.g., due to the weight and constant pressure) and these glasses reducethe peripheral vision for instructions needed for users or reduce visionin general due to glare on the lenses, which may impact both safety andjob accuracy. Existing technology solutions also fail to have morerobust functionality to meet the needs of multiple users for varioustasks. Further, some existing asset handling technologies are based onstatic projection methods that are also burdensome. For example, somesolutions require users to move around a large projector cart or wagonmounted with a generic projector to guide users to destinations.

Various embodiments of the present disclosure improve these existingtechnologies, such as smart glasses, by at least utilizing a hands-freeuser device(s), a control system or server in networked communicationwith the hands-free user device(s), and/or a generated augmented realityenvironment to facilitate handling and transport of an asset or packageby carrier personnel or the like. The handling and/or transport of theasset or package may be related to a picking of the asset from a picklocation (e.g., to “pull” the asset to fulfill an order thereof by acustomer), the sorting of the asset to a sort location (e.g., from aconveyor belt or the like to the next location in which transport orhandling of the asset may occur, for example, on the shelving of awarehouse or a vehicle). The hands-free user device(s) enables carrierpersonnel to transport and/or handle the asset or package in a safe,ergonomic, efficient, and accurate matter, regardless of where (e.g., toand from) the handling and/or transport is occurring, at least within athree dimensional environment mapped via the hands-free user device(s).

In an illustrative example of how these existing technologies areimproved according to aspects of the present disclosure, a user devicecan be worn by a user, such as on a wearable article of clothing, asopposed to placing eyewear over a user's eyes or using a mobile or cartdevice for the handling of assets. After one or more asset identifiersare obtained and location data is determined for an asset, one or morenavigational projections can be dynamically generated and displayed(e.g., within a physical environment a user is in, as opposed to a lensmedium) to guide the user to the location associated with the locationdata. Aspects can also detect handling of the asset by the user (e.g.,via cameras, sensors). One or more notifications associated with thedetection of the handling can be received (e.g., from a control system).Location data can be determined based on analyzing asset identifier dataand analyzing a present position of the user device. Further, a user'senvironment may be mapped based at least on generating amultidimensional graphical representation of the environment andassociating one or more asset locations within the mapped environment.At least each of these new functionalities improve existingtechnologies, as these are functionalities that various existingtechnologies do not now employ.

Conventional methods in the shipping and other industries rely uponcarrier personnel manually reading and/or scanning asset identifier dataassociated with the asset or package and then based thereon manuallytransporting the package or asset to the proper location. A pick orinventory location code or identifier and/or a sort location code oridentifier could then also—in some instances—be read or scanned toconfirm transport was correctly completed. Inefficiencies andinaccuracies are oftentimes encountered.

To address these inefficiencies and inaccuracies of the conventionalmethods, various non-conventional methods have been employed in thepresent disclosure. For example, such non-conventional methods includethe following operations: obtaining one or more asset identifiersdetermining location data for the associated asset(s). Navigationalprojections can be dynamically generated and displayed (e.g., within aphysical environment a user is in, as opposed to a lens medium) to guidethe user to the location associated with the location data. Aspects canalso detect handling of the asset by the user (e.g., via cameras,sensors). One or more notifications associated with the handling can bereceived (e.g., from a control system). Location data can be determinedbased on analyzing asset identifier data and analyzing a presentposition of the user device. Further, a user's environment may be mappedbased at least on generating a multidimensional graphical representationof the environment and associating one or more asset locations withinthe mapped environment. At least each of these new functionalitiesinclude non-conventional functions.

As used herein, an asset may be a parcel or group of parcels, a packageor group of packages, a box, a crate, a drum, a box strapped to apallet, and/or the like. According to standard practices, packages to besorted may be moved along a conveyor belt from some package source to anintake location (e.g., one or more sort employee workstations). A user(e.g., a sort employee or carrier personnel generally) may scan a barcode on the package, or simply reviews information printed on thepackage, and moves that package to an appropriate sort location (e.g., avehicle, a shelf, and/or the like) based on the information provided onthe package or via the barcode scanner. As described herein, embodimentsutilizing a conveyor belt assembly may rely upon an acquisition device(e.g., a stationary imager) positioned above the conveyor, upstream ofthe intake location or sort employee workstations to capture dataassociated with the package. Additional details in this respect may beunderstood with reference to U.S. Ser. No. 15/581,609, the contents ofwhich as are incorporated by reference herein in their entirety.

Via the hands-free user device(s), the carrier personnel or sortemployee may be guided to particular packages to select for transport.Upon the carrier personnel or sort employee picking up the particularpackages they are guided to, the hands-free user device(s) may beconfigured, according to various embodiments, to generate variousprojections, visible to the carrier personnel or sort employee. Thegenerated projections, which may be three-dimensional or two-dimensionalin form, are configured to guide the carrier personnel or sort employeefrom their current location to the appropriate sort location for theparticular package being handled. Upon arrival—via the guidance of thevarious projections—at the appropriate sort location, the hands-freeuser device(s), upon detecting a placement of the particular package mayfurther verify that the placement is correct. If incorrect,notification(s) may be generated, which notifications may take multipleforms, as detailed elsewhere herein.

In the context of picking, the control system may, via the network,interface with the hands-free user device(s) so as to generate one ormore of various projections to guide the carrier personnel or pickemployee to the location of a particular package that needs to be pickedor “pulled” for order fulfillment from a warehouse location or the like.Upon arrival of the carrier personnel or pick employee at the picklocation, specific projections may be generated, so as to advise thepersonnel or employee which specific package should be picked/pulledand/or how many packages (i.e., of the same type) should bepicked/pulled. It should be understood that upon or in response todetection of the picking/pulling of the package(s) by the user device,the latter may be further configured to then guide the carrier personnelto a subsequent location for ongoing handling/transport of the package.Exemplary subsequent locations may include a conveyor belt and/or a sortlocation, as discussed above and also detailed elsewhere herein.

In certain embodiments, although not necessary via the three dimensionalmapping of the facility or warehouse and the network interface betweenthe user device(s) and the control system/server, the hands-free userdevice(s) may utilize software that not only detects changes in handlingof the packages (e.g., picking up or placement actions), but that alsodetects various markers or identifiers distributed throughout thefacility or warehouse, so as to ensure accuracy of the guidance and/ornavigational instructions provided to the carrier personnel. In otherembodiments, no such markers or identifiers may be provided, as thethree dimensional mapping via the user device(s)—with networkingconnectivity to the control system/server—may be utilized to calibrateand establish defined locations (i.e., pick or sort) throughout thefacility or warehouse prior to utilization of the hands-free userdevice(s) for operational purposes by the carrier personnel. In someembodiments, radio signal triangulation (RFID/WIFI), digital compass,and/or any other current method to determine indoor and outdoor positionand bearing may be utilized.

II. Computer Program Products, Methods, and Computing Entities

Embodiments of the present disclosure may be implemented in variousways, including as computer program products that comprise articles ofmanufacture. A computer program product may include a non-transitorycomputer-readable storage medium storing applications, programs, programmodules, scripts, source code, program code, object code, byte code,compiled code, interpreted code, machine code, executable instructions,and/or the like (also referred to herein as executable instructions,instructions for execution, computer program products, program code,and/or similar terms used herein interchangeably). Such non-transitorycomputer-readable storage media include all computer-readable media(including volatile and non-volatile media).

In one embodiment, a non-volatile computer-readable storage medium mayinclude a floppy disk, flexible disk, hard disk, solid-state storage(SSS) (e.g., a solid state drive (SSD), solid state card (SSC), solidstate module (SSM)), enterprise flash drive, magnetic tape, or any othernon-transitory magnetic medium, and/or the like. A non-volatilecomputer-readable storage medium may also include a punch card, papertape, optical mark sheet (or any other physical medium with patterns ofholes or other optically recognizable indicia), compact disc read onlymemory (CD-ROM), compact disc-rewritable (CD-RW), digital versatile disc(DVD), Blu-ray disc (BD), any other non-transitory optical medium,and/or the like. Such a non-volatile computer-readable storage mediummay also include read-only memory (ROM), programmable read-only memory(PROM), erasable programmable read-only memory (EPROM), electricallyerasable programmable read-only memory (EEPROM), flash memory (e.g.,Serial, NAND, NOR, and/or the like), multimedia memory cards (MMC),secure digital (SD) memory cards, SmartMedia cards, CompactFlash (CF)cards, Memory Sticks, and/or the like. Further, a non-volatilecomputer-readable storage medium may also include conductive-bridgingrandom access memory (CBRAM), phase-change random access memory (PRAM),ferroelectric random-access memory (FeRAM), non-volatile random-accessmemory (NVRAM), magnetoresistive random-access memory (MRAM), resistiverandom-access memory (RRAM), Silicon-Oxide-Nitride-Oxide-Silicon memory(SONOS), floating junction gate random access memory (FJG RAM),Millipede memory, racetrack memory, and/or the like.

In one embodiment, a volatile computer-readable storage medium mayinclude random access memory (RAM), dynamic random access memory (DRAM),static random access memory (SRAM), fast page mode dynamic random accessmemory (FPM DRAM), extended data-out dynamic random access memory (EDODRAM), synchronous dynamic random access memory (SDRAM), double datarate synchronous dynamic random access memory (DDR SDRAM), double datarate type two synchronous dynamic random access memory (DDR2 SDRAM),double data rate type three synchronous dynamic random access memory(DDR3 SDRAM), Rambus dynamic random access memory (RDRAM), TwinTransistor RAM (TTRAM), Thyristor RAM (T-RAM), Zero-capacitor (Z-RAM),Rambus in-line memory module (RIMM), dual in-line memory module (DIMM),single in-line memory module (SIMM), video random access memory (VRAM),cache memory (including various levels), flash memory, register memory,and/or the like. It will be appreciated that where embodiments aredescribed to use a computer-readable storage medium, other types ofcomputer-readable storage media may be substituted for or used inaddition to the computer-readable storage media described above.

As should be appreciated, various embodiments of the present disclosuremay also be implemented as methods, apparatus, systems, computingdevices, computing entities, and/or the like. As such, embodiments ofthe present disclosure may take the form of an apparatus, system,computing device, computing entity, and/or the like executinginstructions stored on a computer-readable storage medium to performcertain steps or operations. However, embodiments of the presentdisclosure may also take the form of an entirely hardware embodimentperforming certain steps or operations.

Embodiments of the present disclosure are described below with referenceto block diagrams and flowchart illustrations. Thus, it should beunderstood that each block of the block diagrams and flowchartillustrations may be implemented in the form of a computer programproduct, an entirely hardware embodiment, a combination of hardware andcomputer program products, and/or apparatus, systems, computing devices,computing entities, and/or the like carrying out instructions,operations, steps, and similar words used interchangeably (e.g., theexecutable instructions, instructions for execution, program code,and/or the like) on a computer-readable storage medium for execution.For example, retrieval, loading, and execution of code may be performedsequentially such that one instruction is retrieved, loaded, andexecuted at a time. In some exemplary embodiments, retrieval, loading,and/or execution may be performed in parallel such that multipleinstructions are retrieved, loaded, and/or executed together. Thus, suchembodiments can produce specifically-configured machines performing thesteps or operations specified in the block diagrams and flowchartillustrations. Accordingly, the block diagrams and flowchartillustrations support various combinations of embodiments for performingthe specified instructions, operations, or steps.

III. Exemplary System Architecture

Generally, embodiments of the present disclosure relate to concepts forutilizing a hands-free user device(s), a control system or server innetworked communication with the hands-free user device(s), and agenerated augmented reality environment to facilitate handling andtransport of an asset or package by carrier personnel or the like. FIG.1 is a schematic diagram showing the exemplary communicationrelationships between components of various embodiments of the presentdisclosure. As shown in FIG. 1 , the system may include one or morecontrol systems 100, one or more user devices 110, one or more(optionally) location devices 415 associated with a location 400 (e.g.,a sort location or a pick location), one or more (optionally) conveyorbelt assemblies 800, and one or more networks 105. Each of thecomponents of the system may be in electronic communication with oneanother over the same or different wireless or wired networks including,for example, a wired or wireless Personal Area Network (PAN), Local AreaNetwork (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN),or the like. Additionally, while FIG. 1 illustrates certain systementities as separate, standalone entities, the various embodiments arenot limited to this particular architecture.

A. Exemplary Control System

FIG. 2 provides a schematic of a control system 100 according to oneembodiment of the present disclosure. As described above, the controlsystem 100 may be incorporated into a system as one or more componentsfor providing information regarding the appropriate location 400 foreach of one or more assets 10 (see FIGS. 8-12 ). In general, the termscomputing entity, computer, entity, device, system, and/or similar wordsused herein interchangeably may refer to, for example, one or morecomputers, computing entities, desktops, mobile phones, tablets,phablets, notebooks, laptops, distributed systems, gaming consoles(e.g., Xbox, Play Station, Wii), watches, glasses, key fobs, radiofrequency identification (RFID) tags, ear pieces, scanners, televisions,dongles, cameras, wristbands, kiosks, input terminals, servers or servernetworks, blades, gateways, switches, processing devices, processingentities, set-top boxes, relays, routers, network access points, basestations, the like, and/or any combination of devices or entitiesadapted to perform the functions, operations, and/or processes describedherein. Such functions, operations, and/or processes may include, forexample, transmitting, receiving, operating on, processing, displaying,storing, determining, creating/generating, monitoring, evaluating,comparing, and/or similar terms used herein interchangeably. In oneembodiment, these functions, operations, and/or processes can beperformed on data, content, information, and/or similar terms usedherein interchangeably. The control system 100 may also comprise variousother systems, such as an Address Matching System (AMS), an InternetMembership System (IMS), a Customer Profile System (CPS), a PackageCenter Information System (PCIS), a Customized Pickup and DeliverySystem (CPAD), a Web Content Management System (WCMS), a NotificationEmail System (NES), a Fraud Prevention System (FPS), and a variety ofother systems and their corresponding components.

As indicated, in one embodiment, the control system 100 may also includeone or more communications interfaces 220 for communicating with variouscomputing entities, such as by communicating data, content, information,and/or similar terms used herein interchangeably that can betransmitted, received, operated on, processed, displayed, stored, and/orthe like.

As shown in FIG. 2 , in one embodiment, the control system 100 mayinclude or be in communication with one or more processing elements 205(also referred to as processors, processing circuitry, and/or similarterms used herein interchangeably) that communicate with other elementswithin the control system 100 via a bus, for example. As will beunderstood, the processing element 205 may be embodied in a number ofdifferent ways. For example, the processing element 205 may be embodiedas one or more complex programmable logic devices (CPLDs),microprocessors, multi-core processors, co-processing entities,application-specific instruction-set processors (ASIPs),microcontrollers, and/or controllers. Further, the processing element205 may be embodied as one or more other processing devices orcircuitry. The term circuitry may refer to an entirely hardwareembodiment or a combination of hardware and computer program products.Thus, the processing element 205 may be embodied as integrated circuits,application specific integrated circuits (ASICs), field programmablegate arrays (FPGAs), programmable logic arrays (PLAs), hardwareaccelerators, other circuitry, and/or the like. As will therefore beunderstood, the processing element 205 may be configured for aparticular use or configured to execute instructions stored in volatileor non-volatile media or otherwise accessible to the processing element205. As such, whether configured by hardware or computer programproducts, or by a combination thereof, the processing element 205 may becapable of performing steps or operations according to embodiments ofthe present disclosure when configured accordingly.

In one embodiment, the control system 100 may further include or be incommunication with non-volatile media (also referred to as non-volatilestorage, memory, memory storage, memory circuitry and/or similar termsused herein interchangeably). In one embodiment, the non-volatilestorage or memory may include one or more non-volatile storage or memorymedia 210, including but not limited to hard disks, ROM, PROM, EPROM,EEPROM, flash memory, MMCs, SD memory cards, Memory Sticks, CBRAM, PRAM,FeRAM, NVRAM, MRAM, RRAM, SONOS, FJG RAM, Millipede memory, racetrackmemory, and/or the like. As will be recognized, the non-volatile storageor memory media may store databases, database instances, databasemanagement systems, data, applications, programs, program modules,scripts, source code, object code, byte code, compiled code, interpretedcode, machine code, executable instructions, and/or the like. Such codemay include an operating system, an acquisition module, a sort locationmodule, a matching module, and a notification module. The termsdatabase, database instance, database management system, and/or similarterms used herein interchangeably may refer to a structured collectionof records or data that is stored in a computer-readable storage medium,such as via a relational database, hierarchical database, and/or networkdatabase.

In one embodiment, the control system 100 may further include or be incommunication with volatile media (also referred to as volatile storage,memory, memory storage, memory circuitry and/or similar terms usedherein interchangeably). In one embodiment, the volatile storage ormemory may also include one or more volatile storage or memory media215, including but not limited to RAM, DRAM, SRAM, FPM DRAM, EDO DRAM,SDRAM, DDR SDRAM, DDR2 SDRAM, DDR3 SDRAM, RDRAM, TTRAM, T-RAM, Z-RAM,RIMM, DIMM, SIMM, VRAM, cache memory, register memory, and/or the like.As will be recognized, the volatile storage or memory media may be usedto store at least portions of the databases, database instances,database management systems, data, applications, programs, programmodules, scripts, source code, object code, byte code, compiled code,interpreted code, machine code, executable instructions, and/or the likebeing executed by, for example, the processing element 205. Thus, thedatabases, database instances, database management systems, data,applications, programs, program modules, scripts, source code, objectcode, byte code, compiled code, interpreted code, machine code,executable instructions, and/or the like may be used to control certainaspects of the operation of the control system 100 with the assistanceof the processing element 205 and operating system.

As indicated, in one embodiment, the control system 100 may also includeone or more communications interfaces 220 for communicating with variouscomputing entities, such as by communicating data, content, information,and/or similar terms used herein interchangeably that can betransmitted, received, operated on, processed, displayed, stored, and/orthe like. Such communication may be executed using a wired datatransmission protocol, such as fiber distributed data interface (FDDI),digital subscriber line (DSL), Ethernet, asynchronous transfer mode(ATM), frame relay, data over cable service interface specification(DOCSIS), or any other wired transmission protocol. Similarly, thecontrol system 100 may be configured to communicate via wirelessexternal communication networks using any of a variety of protocols,such as general packet radio service (GPRS), Universal MobileTelecommunications System (UMTS), Code Division Multiple Access 2000(CDMA2000), CDMA2000 1× (1×RTT), Wideband Code Division Multiple Access(WCDMA), Time Division-Synchronous Code Division Multiple Access(TD-SCDMA), Long Term Evolution (LTE), Evolved Universal TerrestrialRadio Access Network (E-UTRAN), Evolution-Data Optimized (EVDO), HighSpeed Packet Access (HSPA), High-Speed Downlink Packet Access (HSDPA),IEEE 802.11 (Wi-Fi), 802.16 (WiMAX), ultra-wideband (UWB), infrared (IR)protocols, near field communication (NFC) protocols, Bluetooth™protocols (e.g., Bluetooth™ Smart), wireless universal serial bus (USB)protocols, and/or any other wireless protocol.

The control system 100 may include or be in communication with one ormore input elements, such as a keyboard input, a mouse input, a touchscreen/display input, motion input, movement input, audio input,pointing device input, joystick input, keypad input, and/or the like.The control system 100 may also include or be in communication with oneor more output elements (not shown), such as audio output, video output,screen/display output, motion output, movement output, and/or the like.

As will be appreciated, one or more of the control system's 100components may be located remotely from other control system 100components, such as in a distributed system. Furthermore, one or more ofthe components may be combined and additional components performingfunctions described herein may be included in the control system 100.Thus, the control system 100 can be adapted to accommodate a variety ofneeds and circumstances. As will be recognized, these architectures anddescriptions are provided for exemplary purposes only and are notlimiting to the various embodiments. Additional details in this respectmay be understood from U.S. Ser. No. 15/390,109, the contents of whichas are incorporated herein by reference in their entirety.

As will also be appreciated, the control system 100 may be generallyconfigured to maintain and/or update a defined location map associatedwith a facility or warehouse in which the user device(s) will beoperated. This may be maintained for provision to the user device(s)upon calibration or initial “environment mapping” (see FIG. 8 ) via theuser device(s); in other embodiments, the control system may maintainthe defined location map—indicating where each package or asset shouldbe located (whether for picking or sorting)—as a fail-safe check orvalidation to be assessed against the environment mapping conducted viaand at the user device(s). In these embodiments, location devices 415(or identifier tags/codes/or the like) may be provided at the respectivelocations, for scanning or recognition via the user device(s) duringcalibration and/or environment mapping. In at least one preferredembodiment, however, the environment mapping occurs without need for orutilization of such location devices 415.

B. Exemplary User Device

FIG. 3 depicts a user device 110 that a user 5 (e.g., user 5 of FIGS.8-12 ) may operate. As used herein, a user may be an individual (e.g.,carrier personnel, such as a sort employee, a pick employee, or thelike), a group of individuals, and/or the like. In various embodiments,the user may operate the user device 110, which may include one or morecomponents that are functionally similar to those of the control system100. In one embodiment, the user device 110 may be one or more mobilephones, tablets, watches, glasses (e.g., Google Glass, HoloLens, VuzixM-100, SeeThru, Optinvent ORA-S, Epson Moverio BT-300, Epson MoverioBT-2000, ODG R-7, binocular Smart Glasses, monocular Smart Glasses, andthe like), wristbands, and the like, and/or any combination of devicesor entities adapted to perform the functions, operations, and/orprocesses described herein. In particularly preferred embodiments hereinthe user device 110 is a hands-free type device, including wearableitems/devices (e.g., the user devices of FIGS. 4-5 ), head-mounteddisplays (HMDs) (e.g., Oculus Rift, Sony HMZ-T3 W, and the like), andthe like. It should be understood that, in at least these embodiments,the user device 110 is configured to not impede the user's range ofvision and/or the like during the use thereof, as is possible in otherembodiments wherein, for example, the user device 110 is a type ofglasses or the like.

It should also be understood that the term user device 110 is intendedto refer to any device that projects, superimposes, overlays, orotherwise provides an image or projection on a surface with respect to auser's viewing angle or line of vision or a user device 110's angle.With reference now to FIGS. 4 and 5 , illustrated therein is anexemplary embodiment of a hands-free user device 110 according tovarious solutions described herein. The device 110 is configured forchest mounting via a mounting mechanism 112 in the illustratedembodiment. The mounting mechanism 112 may include a shoulder strapportion 112-1 configured to adjustably wrap around a user's anterior andposterior portion of the user's shoulders and a waist portion 112-2configured to adjustably wrap around the user's waist. The shoulderstrap portion 112-1 is connected to a top portion of the devicecomponent 114 at the anterior side and a top portion of the waistportion 112-2 at the posterior side. As discussed herein, it should beunderstood that the device 110 may come in any suitable form and bemounted in any suitable manner and in any suitable location. Forexample, in some embodiments, the user device 110 does not include themounting mechanism 112 and may be mounted to a component: on the user'shead (e.g., a hardhat), within a wristband, within a sock, within aglove, within a shirt, and/or within any other suitable article ofclothing in any orientation. It may also be provided with a mountingmechanism (not illustrated) on a picking cart or a forklift or any typeof component operated and/or being moved by the carrier personnel. Insome embodiments, other mounts could place the device 110 on top of thehead (e.g., via a helmet, cap or headband), or shoulder mounted. This,in order to avoid covering the projection element when carrying a parcelin front of the chest. In some embodiments, the projection element, thesensors and the processing units are mounted in different parts of thebody to get a better weight distribution.

Remaining with FIGS. 4 and 5 , the user device 110 in its hands-freeform may include not only the mounting mechanism 112 but also a devicecomponent 114 that together define and constitute the user device 110 insome embodiments. Via the device component 114 (FIG. 5 ), the userdevice 110 in its hands-free form may include an antenna 115 (e.g., theantenna 312 of the user device of FIG. 3 ), a camera 116, aspeaker/microphone 117, a pivoting laser projector 118 and two or morethree-dimensional depth sensors 119. In this respect, it should morebroadly be understood that the term user device 110 is intended to alsoinclude any other peripheral electronics and functionality that may beprovided in conjunction with such devices. For example, the user device110 may include speakers, headphones, or other electronic hardware foraudio output, a plurality of display devices, one or more positionsensors (e.g., gyroscopes, global positioning system receivers, and/oraccelerometers), battery packs, beacons for external sensors (e.g.,infrared lamps), or the like. In one embodiment, the user device 110 canbe used to provide an augmented reality environment/area, a mixedreality environment/area, and/or similar words, as may be used hereininterchangeably, to a user. The terms augmented/mixed environment/areashould be understood to refer to a combined environment/area includingthe physical environment/area and elements of a virtualenvironment/area. In some embodiments, the pivoting laser projector 118is alternatively an LED picoprojector.

In some embodiments, the device component 114 alternatively oradditionally includes different sensors for various functions, such asone or more digital compasses, accelerometers and/or gyroscopesconfigured to determine changes in position or speed of a user such thatthe pivoting laser projector 118 projects the correct image in thecorrect orientation. For example, if the user is hanging in a sidewaysmanner, an accelerometer can detect that the associated device component114 is likewise oriented. This information can be identified by aprocessor, which causes the pivoting laser projector 118 to responsivelytransmit a projection in a sideways manner, as opposed to a mannerassociated with the user standing on his/her feet. In another example,the user can be running or otherwise moving at a particular speed, whichcauses the projector 118 to make projections faster/slower based on thespeed or acceleration a user is moving at. Additionally oralternatively, these movement sensors can be used for notificationpurposes to the control system 100. For example, the accelerometer mayinfer that a person is in a particular orientation. These accelerometerreadings may then be transmitted, via the antenna 115, to the controlsystem 100 such that the control system 100 responsively transmits anotification back to the device component 114 in order to warn or notifythe user whether the user is in a suitable orientation. Other sensorsmay be used alternatively or additionally, such as range finders toidentify how far away the device component 114 is from obstacles (e.g.conveyor devices) within an environment. This may help the projectedimage be projected in a more precise three-dimensional manner. Forexample, referring to FIG. 12 , the projection 810 may be projected inits specific orientation based at least on one or more range findersidentifying the precise distance between a user device and the conveyingmechanism 802. In another example, proximity-based sensors (e.g., RFIDreader and tag) may be utilized within the user device and an object(e.g., an asset and/or conveying mechanism) in order to trigger theappropriate projections projected by the projector 118. For example, thedevice component 114 may include a tag reader that, when within aproximity or signal strength threshold of a tag located on anasset/object, triggers projections and/or notifications from the controlsystem 100. In another example, assets or other pieces of equipment(e.g., a conveyance mechanism) includes one or more beacons configuredto transmit location identifiers to any listening device within athreshold distance. In these embodiments, the listening device may bethe device component 114, which receives the location identifiers andtransmits them to the control system 100 such that the control system100 provides responsive notifications back to the device component 114,such as “pick package Y from shelf X,” etc. In some aspects, the devicecomponent 114 includes one or more location sensors (e.g., beacons, GPSmodules) for the determining and analyzing location data as describedherein.

Referring back to FIG. 3 , the user device 110 can include an antenna312 (e.g., the antenna 115 of FIG. 5 ), a transmitter 304 (e.g., radio),a receiver 306 (e.g., radio), and a processing element 308 (e.g., CPLDs,microprocessors, multi-core processors, co-processing entities, ASIPs,microcontrollers, and/or controllers) that provides signals to andreceives signals from the transmitter 304 and receiver 306,respectively. Certain embodiments of the user device 110 may alsoinclude and/or be associated with any of a variety of sensors (e.g.,three-dimensional sensors, such as the depth sensors 119 of FIG. 5 ),depth cameras (e.g., the camera 116 of FIG. 5 ), three-dimensionalscanners, binocular cameras, stereo-vision systems, pivoting projectors(e.g., the laser projector 118 of FIG. 5 ).

In certain embodiments, the three-dimensional sensors (e.g., sensors 119of FIG. 5 ) and/or the three-dimensional scanners may be utilized to“read” the environment surrounding the user device 110, as detailedelsewhere herein. In those and additional embodiments, the sensorsand/or scanners may build therefrom a three-dimensional model of thearea through which the device 110 travels and/or has travelled. Thisgenerated model, as detailed elsewhere herein, may then be compared byone or more processors within the device 110 to a memory-based map ofthe facility or area (i.e., the environment). By doing so, the scannerreadings may be used to determine which area of the map is in front of auser of the user device 110 (during operation) and extrapolate from thesame the position and heading of the user for future movement.

In certain embodiments, the pivoting projectors may be the pivotinglaser projector 118 of FIG. 5 ; although, in other embodiments theprojectors need not necessarily be laser-based. In various embodiments,it should be understood that the projectors are configured to generateand provide—in a manner visible to the user (e.g., the carrierpersonnel)—one or more navigational guidance projections (e.g., arrows,frames, text, and/or the like). These projections, as will be discussedin further detail elsewhere herein, may be two-dimensionalrepresentations (e.g., as illustrated in FIGS. 8-11C), three-dimensionalrepresentations (e.g., the projection 810 of FIG. 12 ), and/orholographic-based projections, or the like. The projections may beprovided on a floor surface, a wall surface, and/or on or adjacent ashelving structure, as detailed elsewhere herein. Optionally, a separatesensor, a regular camera, or the like (e.g., the camera 116 of FIG. 5 )may also be provided on the user device 110 for reading of the projectedimage(s) and therefrom verify if the projector is working properlyand/or whether the result projected is readable. Via the user device 110and software associated therewith, in conjunction with locationdetermining capabilities of the system described herein generally, theprojections provided via the user device are updated in real-time ornear-real-time, as the user moves physically. A refresh rate in therange of 35-60 times per second may be provided, although differingrates of refreshing may be desirable, provided that the rate provided issubstantially real-time or near real-time in nature.

Remaining with FIG. 5 , the camera 116 of the user device component 114illustrated therein in its hands-free form may be utilized as afail-safe for visual confirmation or the like of correct/accuratehandling of an asset or package by the user of the user device 110. Forexample, in some embodiments the camera 116 captures each assetidentifier and/or asset location identifier as a user traverses anenvironment. This location data and asset identifier data may then betransmitted, in near-real time via the antenna 115, to the controlsystem 100. The control system 100 may then compare the asset identifierto asset identifiers stored in a data store to identify an asset and dothe same with the captured location. In this way, the control system 100may identify any discrepancies between the asset and the location bylocating any mismatches between identifiers. For example, the controlsystem 100 may determine that the identifier associated with package Xshould be located, picked, and/or placed at shelf Y, but the camera 116captured it located in, picked, sorted and/or placed at shelf B. Anotification indicating this may be responsively transmitted back to thedevice 114 such that the speaker 117 issues a prompt indicating thediscrepancy and/or telling the user where the correct location is forthe particular package. In some embodiments, the device component 114itself determines this information without the need to transmit theinformation to the control system 100 for processing. In someembodiments, other notifications may be provided additionally oralternatively, such as a visual notification within a display screen onthe device component and/or a notification that causes vibration of thedevice component 114.

In certain embodiments, the camera 116 may be utilized as a verificationmechanism for ensuring that the projector 118 is working properly and/oris displaying readable projections. For example, the device component114 may stream in near-real-time information captured via the camera 116to the control system 100. If no projections are captured, this maytrigger an alert (e.g., to a supervisor mobile device), which indicatesthat the projections are not being made. Likewise, if a projection isnot verified (e.g., because there is a lot of light reducing projectionimage boundaries), a notification can be made in a similar manner asdescribed above. The speaker 117 may be utilized in conjunctiontherewith, so as to provide audible commands to the user (e.g.,delivered from the control system 100 to the component 114 via theantenna 115) should a deviation occur and/or to enable the user of theuser device to communicate, via the network, with the control system ina near real-time or real-time manner. For example, in some embodiments,the speaker 117 alternatively or additionally includes a microphone thatpicks up sound variations that are stored in the memory. The soundvariations may correspond to a command or natural language phrase issuedby the user, such as “where do I find item X?” or “where is shelf Ylocated?” Responsively, these sound variations are transmitted, via theantenna 115, to the control system 100. In these embodiments, thecontrol system 100 may employ one or more voice recognition algorithmsto interpret the sound variations and provide one or more responsivenotifications back to the device component 114, such that the speaker117 provides the notification output. For example, in response to theuser question of “where do I find item X?” the control system 100 mayinterpret the phrase, identify a data structure that associates thelocation with item X. The control system 100 may then responsivelytransmit to the device component 114 a notification that causes thespeaker 117 to output the location of where item X is.

Returning back to FIG. 3 , the signals provided to and received from thetransmitter 304 and the receiver 306, respectively, may includesignaling information in accordance with air interface standards ofapplicable wireless systems. In this regard, the user device 110 may becapable of operating with one or more air interface standards,communication protocols, modulation types, and access types. Moreparticularly, the user device 110 may operate in accordance with any ofa number of wireless communication standards and protocols, such asthose described above with regard to the control system 100. In aparticular embodiment, the user device 110 may operate in accordancewith multiple wireless communication standards and protocols, such asUMTS, CDMA2000, 1×RTT, WCDMA, TD-SCDMA, LTE, E-UTRAN, EVDO, HSPA, HSDPA,Wi-Fi, WiMAX, UWB, IR, NFC, Bluetooth™ Smart, USB, and/or the like.Similarly, the user device 110 may operate in accordance with multiplewired communication standards and protocols, such as those describedabove with regard to the control system 100 via a network interface 320.

Via these communication standards and protocols, the user device 110 cancommunicate with various other entities (e.g., the control system 100, alocation device 415, or the like) using concepts such as UnstructuredSupplementary Service Data (USSD), Short Message Service (SMS),Multimedia Messaging Service (MMS), Dual-Tone Multi-Frequency Signaling(DTMF), and/or Subscriber Identity Module Dialer (SIM dialer). The userdevice 110 can also download changes, add-ons, and updates, forinstance, to its firmware, software (e.g., including executableinstructions, applications, program modules), and operating system; thismay occur periodically, upon initiation via a user of the user device110, or upon cue(s) received at the user device from the control system100.

According to one embodiment, the user device 110 may also include alocation and/or perspective determining aspect, device, module,functionality, and/or similar words used herein interchangeably. Forexample, the user device 110 may include outdoor and/or environmentalpositioning aspects, such as a location module adapted to acquire, forexample, latitude, longitude, geocode, course, direction, heading,speed, universal time (UTC), date, and/or various otherinformation/data. In one embodiment, the location module can acquiredata, sometimes known as ephemeris data, by identifying the number ofsatellites in view and the relative positions of those satellites. Thesatellites may be a variety of different satellites, including Low EarthOrbit (LEO) satellite systems, Department of Defense (DOD) satellitesystems, the European Union Galileo positioning systems, the ChineseCompass navigation systems, Indian Regional Navigational satellitesystems, and/or the like. Alternatively, the location information may bedetermined by triangulating the user device 110's position in connectionwith a variety of other systems, including cellular towers, Wi-Fi accesspoints, and/or the like. Similarly, the user device 110 may includeindoor positioning aspects, such as a location/environment moduleadapted to acquire, for example, latitude, longitude, geocode, course,direction, heading, speed, time, date, and/or various otherinformation/data. Some of the indoor systems may use various position orlocation technologies including RFID tags, indoor beacons ortransmitters, Wi-Fi access points, cellular towers, nearby computingdevices (e.g., smartphones, laptops), nearby components with knownrelative locations, and/or the like. For instance, such technologies mayinclude the iBeacons, Gimbal proximity beacons, Bluetooth Low Energy(BLE) transmitters, Near Field Communication (NFC) transmitters,three-dimensional scanners, robot vision systems, environmental mappingdevices, and/or the like. These indoor positioning aspects can be usedin a variety of settings to determine the location of someone orsomething to within inches or centimeters.

The user device 110 may also detect markers and/or target objects. Forexample, the user device 110 may include readers, scanners, cameras,sensors, and/or the like for detecting when a marker and/or targetobject and/or a pattern of unique colors within its point-of-view(POV)/field-of-view (FOV) of the real world environment/area. Forexample, readers, scanners, cameras, sensors, and/or the like mayinclude RFID readers/interrogators to read RFID tags, scanners andcameras to capture visual patterns and/or codes (e.g., text, barcodes,character strings, Aztec Codes, MaxiCodes, information/data Matrices, QRCodes, electronic representations, and/or the like), and sensors todetect beacon signals transmitted from target objects or theenvironment/area in which target objects are located. For example, insome embodiments, the user device 110 may detect signals transmittedfrom the control system 100 (FIGS. 1-2 ), an asset 10 (FIG. 8 ), animproved conveyor belt assembly (FIG. 12 ), and/or from a locationdevice 415 (FIG. 1 ), as may be desirable or advantageous.

In one embodiment, the user device 110 may include accelerometercircuitry for detecting movement, pitch, bearing, orientation, and thelike of the user device 110. This information/data may be used todetermine which area of the augmented/mixed environment/area correspondsto the orientation/bearing of the user device 110 (e.g., x, y, and zaxes), so that the corresponding environment/area of the augmented/mixedenvironment/area may be displayed via the display along with a displayedimage. For example, the user device 110 may overlay an image in aportion of the user's POV/FOV of the real world environment/area. Inthese and other embodiments, the user device 110 may also includecircuitry and/or software for determining when a change in the handlingof a package or asset by a user of the user device has occurred.Exemplary changes detected may include the picking up of an asset orpackage, the setting down of an asset or package, or the like.

The user device 110 may also comprise or be associated with an assetindicia reader, device, module, functionality, and/or similar words usedherein interchangeably. For example, the user device 110 may include acamera or RFID tag reader configured to receive information from passiveRFID tags and/or from active RFID tags associated with an asset 10. Theuser device 110 may additionally or alternatively include an opticalreader configured for receiving information printed on an asset 10. Forexample, the optical reader may be configured to receive informationstored as a bar code, QR code, or other machine-readable code. Theoptical reader may be integral to the user device 110 and/or may be anexternal peripheral device in electronic communication with the userdevice 110. The optical reader may also or alternatively be configuredto receive information stored as human readable text, such ascharacters, character strings, symbols, and/or the like. The user device110 may utilize the asset indicia reader to receive informationregarding an asset 10 to be sorted.

In at least one embodiment, the user device 110 may be equipped with anoptical reader or the like configured to receive and/or monitorinformation associated with an associated conveyor belt, as detailedelsewhere herein. For example, the optical reader may be configured toreceive and/or otherwise monitor and/or recognize a pattern located onthe conveyor belt and associated with respective assets or packages.Additional details in this respect may be understood with reference toU.S. Ser. No. 15/581,609, the contents of which as are incorporated byreference herein in their entirety

The user device 110 may also comprise a user interface (that can includea display or see-through display 314 coupled to a processing element 308and/or a user input device 318 coupled to a processing element 308). Forexample, the user interface may be a user application, browser, userinterface, and/or similar words used herein interchangeably executing onand/or accessible via the user device 110 to interact with and/or causedisplay of information, as described herein. The user interface cancomprise any of a number of devices allowing the user device 110 toreceive data, such as a keypad (hard or soft), a touch display, voice ormotion interfaces, or other input device. In embodiments including akeypad, the keypad can include (or cause display of) the conventionalnumeric (0-9) and related keys (#, *), and other keys used for operatingthe user device 110 and may include a full set of alphabetic keys or setof keys that may be activated to provide a full set of alphanumerickeys. In addition to providing input, the user input interface can beused, for example, to activate or deactivate certain functions, such asscreen savers and/or sleep modes.

The user device 110 can also include volatile storage or memory 322and/or non-volatile storage or memory 324, which can be embedded and/ormay be removable. For example, the non-volatile memory may be ROM, PROM,EPROM, EEPROM, flash memory, MMCs, SD memory cards, Memory Sticks,CBRAM, PRAM, FeRAM, NVRAM, MRAM, RRAM, SONOS, FJG RAM, Millipede memory,racetrack memory, and/or the like. The volatile memory may be RAM, DRAM,SRAM, FPM DRAM, EDO DRAM, SDRAM, DDR SDRAM, DDR2 SDRAM, DDR3 SDRAM,RDRAM, TTRAM, T-RAM, Z-RAM, RIMM, DIMM, SIMM, VRAM, cache memory,register memory, and/or the like. The volatile and non-volatile storageor memory can store databases, database instances, database managementsystems, data, applications, programs, program modules, scripts, sourcecode, object code, byte code, compiled code, interpreted code, machinecode, executable instructions, and/or the like to implement thefunctions of the user device 110. As indicated, this may include a userapplication that is resident on the entity or accessible through abrowser or other user interface for communicating with the controlsystem 100 (FIG. 2 ), location device 415 (FIG. 1 ), and/or variousother computing entities.

In another embodiment, the user device 110 may include one or morecomponents or functionality that are the same or similar to those of thecontrol system 100, as described in greater detail above. As will berecognized, these architectures and descriptions are provided forexemplary purposes only and are not limiting to the various embodiments.

In the embodiment shown in FIGS. 4 and 5 , an information gatheringdevice may be provided via a combination of the image camera 116 that ismounted on the device component 114 and/or the three-dimensional depthsensors 119. In other embodiments, the information gathering device maybe a three-dimensional depth sensor, a stereo camera, and/or thelike—utilized independently relative to the camera. In all of these andadditional embodiments, the displayed or captured image data (e.g.,FIGS. 8-11 described elsewhere herein) is merged with objects in thephysical world/environment in a seamless manner, so as to provide asense that the displayed image(s) or projection is an extension of thereality present in the physical world/environment. This is oftentimesreferred to as a “mixed reality” or a “hybrid reality” environment,whereby the merging of real and virtual worlds produces a newenvironment containing visualizations of both physical and digitalobjects that are able to co-exist and interact relative to one anotherin a real-time manner. Stated otherwise, provided and/or generated is anoverlay of synthetic content on the real world or physical environment,with the former being anchored to and able to in a real-time manner(e.g., upon movement of a user) interact with the real world or physicalenvironment.

In the exemplary embodiment of FIGS. 4 and 5 , the overlay provided viathe user device 110 and its physical component 114 is at least a twodimensional representation that is projected ahead of or before the userof the user device, so as to provide handling/movement guidance (i.e.,navigational guidance) for the user during transport of or travel toinitiate transport of an asset or package.

C. Exemplary Conveyor Belt Assembly

FIG. 12 depicts a conveyor belt assembly 800 in communication with thecontrol system 100, where the improved conveyor belt assemblyfacilitates obtaining of asset 10 information. In the embodimentdepicted in FIG. 12 , the conveyor belt assembly 800 may comprise aconveying mechanism 802 and an acquisition/display entity 804 (forcapturing the asset 10 information), each of which as are described infurther detail in U.S. Ser. No. 15/581,609, the contents of which as areincorporated by reference herein in their entirety.

Of note relative to FIG. 12 , in conjunction with the hands-free userdevice(s) 110 described elsewhere herein, a user 5 may be guided towarda particular asset or package on the conveying mechanism 802 via one ormore navigational projections 810. In at least the illustratedembodiment, the projections 810 are provided in a three-dimensionalform, as may be compared with the two-dimensional projections in FIGS.8-11C. Either may be utilized interchangeably, as may beholographic-based projections or the like. Again, as mentioned,additional detail in this respect may be obtained from U.S. Ser. No.15/581,609, the contents of which as are incorporated by referenceherein in their entirety. In various embodiments, a hologram orholographic-based projection is a recording of a light field, as opposedto an image formed by a lens (e.g., of a pair of smart glasses), and isused to display a fully three or more dimensional image without the useor aid of special glasses or other intermediate objects. A hologram canbe displayed within any physical geographical environment without theneed of any projecting medium (e.g., projector screen, object, lens). Insome embodiments, the navigation projection 810 is or includes amultidimensional image that represents a volumetric display, which is avisual representation of an object in at least three physicaldimensions, as opposed to simulating depth or multiple dimensionsthrough visual effects. In these embodiments, the same projected object(e.g., an arrow pointing to an asset over a conveyor) looks differentfrom various perspectives (e.g., a side view, versus a front view,versus a back view). For example, a first front view can include a firstarrow and first instructions for worker X to pick/sort from. From asecond rear view, the same first arrow can include second instructionsfor worker Y to pick/sort from, etc.

D. Exemplary Location Device

In various embodiments, one or more locations 400 (and/or 1400) may beassociated with one or more (optionally provided) location devices 415,with both being configured for identifying one or more assets 10 beingsorted to each location 400. As non-limiting examples, such locations400 may include one or more vehicles (e.g., aircraft, tractor-trailer,cargo container, local delivery vehicles, and/or the like), pallets,identified areas within a building, bins, chutes, conveyor belts,shelves, and/or the like. The locations may be sort locations (fortransport of the asset for additional movement/handling) or picklocations (for storing of the asset until it needs to be picked or“pulled” for order fulfillment purposes of the like). The one or morelocation devices 415 (e.g., 415-1 of FIG. 13E) may be attached to alocation 400 and/or located more generally within and/or at the location1400 (see FIGS. 13A-13F). Alternatively the one or more location devices415 may be located adjacent to a sort location 400/1400 or otherwiseproximate the sort location 400/1400. In various embodiments, a locationdevice 415 may be located proximate to an area designated to store thesort location 400/1400. For example, when the sort location 400 includesa delivery vehicle, a location device 415 may be located above each of aplurality of parking areas designated for one or more delivery vehicles.This may apply equally relative to sort and/or pick locations (e.g.,FIGS. 13D-F).

In various embodiments, the one or more location devices 415 may includecomponents functionally similar to the control system 100 and/or theuser device 110. As noted above in referencing the control system 100,the term “computing entity” may refer to, for example, one or morecomputers, computing entities, desktops, mobile phones, tablets,phablets, notebooks, laptops, distributed systems, gaming consoles(e.g., Xbox, Play Station, Wii), watches, glasses, key fobs, RFID tags,ear pieces, scanners, televisions, dongles, cameras, wristbands, kiosks,input terminals, servers or server networks, blades, gateways, switches,processing devices, processing entities, set-top boxes, relays, routers,network access points, base stations, the like, and/or any combinationof devices or entities adapted to perform the functions, operations,and/or processes described herein Like the user device shownschematically in FIG. 3 , the location device 415 can include anantenna, a transmitter (e.g., radio), a receiver (e.g., radio), and aprocessing element (e.g., CPLDs, microprocessors, multi-core processors,co-processing entities, ASIPs, microcontrollers, and/or controllers)that provides signals to and receives signals from the transmitter andreceiver, respectively.

The signals provided to and received from the transmitter and thereceiver, respectively, may include signaling information in accordancewith air interface standards of applicable wireless systems. In thisregard, the location device 415 may be capable of operating with one ormore air interface standards, communication protocols, modulation types,and access types. More particularly, the location device 415 may operatein accordance with any of a number of wireless communication standardsand protocols, such as those described above with regard to the controlsystem 100. In a particular embodiment, the location device 415 mayoperate in accordance with multiple wireless communication standards andprotocols, such as UMTS, CDMA2000, 1×RTT, WCDMA, TD-SCDMA, LTE, E-UTRAN,EVDO, HSPA, HSDPA, Wi-Fi, WiMAX, UWB, IR, NFC, Bluetooth™, USB, and/orthe like. Similarly, the location device 415 may operate in accordancewith multiple wired communication standards and protocols, such as thosedescribed above with regard to the control system 100 via a networkinterface.

Via these communication standards and protocols, the location device 415can communicate with various other entities (e.g., the user device 110and/or the control system 100) using concepts such as USSD, SMS, MMS,DTMF, and/or SIM dialer. The location device 415 can also downloadchanges, add-ons, and updates, for instance, to its firmware, software(e.g., including executable instructions, applications, programmodules), and operating system. Of course, it should also be understood,as mentioned previously herein, that certain embodiments utilizing thehands-free form of the user device 110 may not utilize any locationdevices 415, whereby location of the various assets (for sorting orpicking) may be communicated directly as between the user device 110 andthe control system 100, further in conjunction with an environmentalmapping capability of the user device 110, as described further below.

E. Exemplary Location

Referring to FIGS. 11A-C and 13D-E, an exemplary location 400 (and/or1400) is schematically depicted. As described above, the location 400may include may include one or more vehicles (e.g., aircraft,tractor-trailer, cargo container, local delivery vehicles, and/or thelike), pallets, identified areas within a building, bins, chutes,conveyor belts, shelves, and/or the like. In the embodiment depicted inthe figures listed above, the location 400 includes a plurality ofshelves onto which the assets 10 may be placed and/or removed from.While these figures depict a specific quantity of shelves as beingstacked in a vertical direction, it should be understood that anyquantity of shelves may be arranged in any suitable configuration tohold the assets 10. Each of the shelves may include one or more visualindicators (e.g., 715 FIG. 10 ) positioned on or proximate to theshelves; however, as mentioned previously herein, certain hands-freeuser device embodiments may dispense with such indicators 110 c.

In these and other embodiments, navigational projections 715, much likethe navigational projections 810 provided in conjunction with theconveyor belt assembly 800 may assist in identifying an appropriateposition for placement and/or removal of the asset 10 within the sortand/or pick location. In particular embodiments, for example, a user 5(FIG. 12 ) may utilize the indicia reader or camera of the user device110 to scan, read, or otherwise receive asset identifier data from theasset 10 to identify, in cooperation with the control system 100, anappropriate position for placement of the asset 10 within the warehouseor facility, namely at the location 400/1400. In other embodiments, thecontrol system 100 may determine the appropriate position for placementof the asset within the warehouse relative to the location 400/1400 andconvey that information to the user device 110 in response to the userdevice having approached an asset or package (e.g., for sorting). Inother embodiments, the control system 100 may proactively transmitprojections to the user device 110 upon receipt of a package or assetorder, requiring “picking” of the asset or package from a pick locationfor order fulfillment purposes or the like. Still further, the controlsystem 100 may determine the appropriate position for placement of theasset 10 within the location 400/1400 based on a variety of factors. Forexample and without limitation, the control system 100 may determine theappropriate position for placement of the asset 10 within the location400 based on the destination of the assets 10.

IV. Exemplary Control System Configuration

In various embodiments, the control system 100 may comprise a pluralityof modules, each module configured to perform at least a portion of thefunctions associated with the methods described herein. For example, thecontrol system 100 may comprise an acquisition module, a locationmodule, and a notification module. Although described herein as beingindividual components of the control system 100, the various modules mayoperate on a combination of one or more devices (e.g., the user device110, the acquisition/display entity 804 (for capturing the asset 10information), the location device 415 (where provided), and/or thecontrol system 100), such that each device performs the functions of oneor more modules.

A. Acquisition Module

In various embodiments, the acquisition module may be configured toobtain asset identifier data associated with an asset 10 to be sortedand/or picked. This asset identifier data may be obtained, in part, viaan order placed by a customer desiring transport and delivery (e.g.,picking, as a first step) of the asset or package. In other embodiments,the asset identifier data may be obtained, in part, via theacquisition/display entity 804 associated with a conveyor belt of thelike, transporting packages or assets to a sort location.

In various embodiments, the asset identifier data may comprise a uniqueasset identifier such as a tracking number or code, and data definingthe one or more appropriate locations 400 for the asset 10 as it movesbetween an origin and a destination, and/or the like.

As a non-limiting example, the acquisition module may be configured toobtain data from the user device 110 (e.g., of FIGS. 3 and 4 ) and/orthe acquisition device 810 (e.g., of FIG. 12 ). In various embodiments,the data received from the user device 110 and/or the acquisition device810 may include the entirety of the asset identifier data and thereforethe acquisition module need only receive asset identifier data from oneof the user device 110 and/or the acquisition device 810. However, invarious embodiments, the data received from the user device 110 (FIGS. 3and 4 ) and/or the acquisition device 810 (FIG. 12 ) may comprise only aportion of the asset identifier data, and the acquisition module may beconfigured to obtain the remainder of the asset identifier data from oneor more other sources. As another non-limiting example, the acquisitionmodule may be configured to search one or more databases incommunication with the control system 100 for asset identifier datacorresponding to the data received from the user device 110 and/or theacquisition device 810. The acquisition module may additionally oralternatively be configured to receive and store at least a portion ofthe asset identifier data corresponding to the asset 10 that is storedin one or more databases.

In various embodiments, the acquisition module may be configured totransmit at least a portion of the asset identifier data to one or moredevices (e.g., the user device 110) and/or one or more modules (e.g.,the location module and/or the notification module). Moreover, uponreceiving the asset identifier data regarding an asset 10 to be sorted,the acquisition module may be configured to link or otherwise associatethe user device 110 and the asset identifier data. As will be describedin greater detail herein, the user device 110 may be associated with theasset identifier data by storing at least a portion of the assetidentifier data in a memory associated with the user device 110.

B. Location Module

The location module may be configured to receive asset identifier datafrom the acquisition module. The sort location module is configured toascertain the appropriate location 400 and/or the appropriate positionwithin the location 400 for the asset 10 based at least in part on theasset identifier data. In certain embodiments, the location module maybe configured to determine the appropriate location 400 based at leastin part on the asset identifier data and location data that isassociated with the each of the plurality of locations 400. The locationdata may be generated based not only upon the asset identifier data, butalso upon the environmental mapping conducted via the user device, asdescribed elsewhere herein.

In various embodiments, each of the plurality of locations 400 may beidentified by location data, which may include a unique locationidentifier. The unique location identifier may comprise a uniquecharacter string individually identifying each of the plurality oflocations 400. In various embodiments, the location data may define anysubsequent processing to be performed on assets 10 within each location400 and/or 1400, and may comprise the unique sort location identifierfor each of the plurality of locations 400/1400 the assets 10 will passthrough. In various embodiments, the location module may determinewhether the processing to be performed on assets 10 in each of theplurality of locations 400 (as defined in the location data) will movethe asset 10 closer to its final destination.

In various embodiments, the location module may determine whether theprocessing steps to be performed on the assets 10 in each of thelocations 400/1400 complies with the service level (e.g., Same Dayshipping, Next Day Air, Second Day Air, 3 Day Select, Ground shipping,and/or the like) corresponding to the asset 10. As a non-limitingexample, the location module may determine the appropriate location foran asset 10 to be delivered to 123 Main Street, Atlanta, Ga. is adelivery vehicle that will deliver other assets 10 to the same addressor nearby addresses (e.g., along the same delivery route). As a secondnon-limiting example, the location module may determine the appropriatelocation for an asset 10 to be delivered to 345 Broad Street, LosAngeles, Calif. via Next Day Delivery is a pallet to be loaded onto aplane destined for Los Angeles, Calif. As yet another non-limitingexample, the location module may determine the appropriate location foran asset 10 prior to its fulfilment for delivery, which location may becharacterized—as done elsewhere herein—as a pick location for the asset.

After determining the appropriate location 400/1400 and/or theappropriate position for the asset 10 within the location 400/1400, thelocation module may be configured to transmit data defining theappropriate location 400/1400 and/or the appropriate position for theasset 10 within the location 400/1400 to one or more devices (e.g., theuser device 110) and/or modules (e.g., the notification module).Additional details in this respect are provided in U.S. Ser. No.15/390,109, the contents of which as are hereby incorporated byreference in their entirety.

C. Notification Module

In various embodiments, the notification module may receive dataindicating whether the location 400 and/or 1400 (e.g., as transmitted tothe control system 100 via the user device) is the appropriate sort orpick location (e.g., as determined by the control system 100) for theasset or package being handled. As described herein, the notificationmodule may cause one or more alerts to be generated in order to notifythe user 5 (e.g., sort or pick personnel, more generally the carrierpersonnel) whether the asset 10 should be deposited in the location 400and/or picked therefrom, however as the case may be. For example, thenotification module may be configured to transmit confirmation dataand/or mistake data to the user device 110 in order to cause the deviceto generate an alert discernible by the user 5 (e.g., carrier personnel)indicative of the appropriate sort location for the asset 10. Toascertain whether confirmation data and/or mistake data is appropriatefor transmission, the user device 110 (and/or sensors associatedtherewith, e.g., three-dimensional sensors) may be configured todetermine not only the position of the asset but also the position ofthe user's hands (e.g., including not only location, but also gestures),so as to gauge whether or not sorting and/or picking of the asset isproceeding properly. For example, the camera 116 may utilize objectrecognition algorithms that identify whenever a person is clasping anobject in a particular manner to determine properness.

In various embodiments, the notification module may cause the userdevice 110 to audibly provide the user with a confirmation message(e.g., via the speaker 117) upon a determination that the location400/1400 is the appropriate sort or pick location. In variousembodiments, the notification module may alternatively or additionallycause one or more sounds to be generated, one or more lights toilluminate, one or more mechanical assemblies to move, and/or otherprocesses discernible by a user 5 to operate and thus indicate to theuser 5 whether the location 400/1400 is the appropriate location. Itshould also be understood that notifications may be generated—andcommunicated to the user via the user device—not only when the user isat the location (e.g., for picking or sorting), but also during travelof the user to/from the location relative to other locations in thewarehouse or facility. As a non-limiting example, with reference to FIG.9 , as the user travels down the hallway or open path, were the user tonavigate contrary to the navigational projection 710 provided, the userdevice could generate an audible (or other type) notification to theuser—either independently, or upon cue received via the control system100. In certain embodiments, notifications may be generated andcommunicated (e.g., via the network or otherwise) to one or more partiesother than the user of the user device (e.g., carrier supervisorypersonnel, other internal carrier personnel (e.g., quality assurancerepresentatives, or the like), external personnel, external third partyentities, or the like). Any of the notifications and/or communicationsdescribed herein may be so communicated, whether to the user aloneand/or to parties other than the user and/or to a combination of both,as may be desirable.

Moreover, the notification module may be configured to generate an alertafter associating asset identifier data with location data and/or cueingan asset or package for picking. The notification module may beconfigured to generate an alert to inform the user 5 (e.g., carrierpersonnel) or other users regarding asset identifier data beingassociated with location data and/or the immediate need for navigationor travel to occur toward the location for picking of the asset orpackage or otherwise.

According to various embodiments, whether adjacent a location 400/1400or a conveying mechanism 802, the notification module may be configuredto generate one or more navigational projections (e.g., 710, 715, 1401and/or the like, with reference to FIGS. 8-11C by way of non-limitingexample) to convey navigational instructions to the user 5. It should beunderstood that according to various embodiments, the navigationalprojections may be computer-generated and/or overlaid over an augmentedreality environment, which may in certain embodiments be displayed tothe user via the utilized user devices 110. In at least the hands-freeembodiment, the navigational projections may be generated via thepivoting laser projector (e.g., 118 of FIG. 5 ) of the user device 110.FIGS. 8-11C and also FIGS. 13A-13F illustrate various types ofnavigational projections as may be generated via the notificationmodule, when conveyed via the control system 100 further to the userdevice 110. In at least one embodiment, it may be understood that thenavigational projections may be generated at/by the user device 110,independent of the control system 100, upon receipt from the controlsystem of only a new “pick” or “sort” command for a particular asset orpackage. In various embodiments, the text indicia and navigationalprojections of FIGS. 13A-13F occur via one or more components of thedevice component 114 (e.g., laser projector 118 to project the arrow1303 and the indicia 1309 of FIG. 13A).

FIG. 13A includes the environment 800 that the user is physicallylocated in, which includes the conveyance device 1305, an asset 1301,various location devices, one of which is indicated by the device 1307.As described above, the location devices help map the environment 800.The text indicia (which may also be considered a navigationalprojection) 1309 may be projected within the environment 800, whichcommands the user to “push forward” the asset 1301. The navigationalprojection 1303 (i.e., the arrow) also assists or illustrates thedirection in which the user should push the asset 1301 forward on theconveyance mechanism 1305. FIG. 13B includes the same environment 800,except a different asset 1315 arrives and the text indicia 1311 commandsthe user to “push” the asset 1315 to the “other side.” The navigationprojection 1313 helps guide the user to show which direction to push theasset 1315. In this way, these visual frames together help guide andinstruct the user for handling of assets. Turning to FIG. 13C, anotherasset 1317 arrives and the text indicia 1319 prompts the user to “pickand sort” the asset 1317 to pick up, sort, and place within a sortinglocation.

FIG. 13D illustrates prompting the user to sort the asset in aparticular location within the environment 1400. In some embodiments,the instructions illustrated in FIG. 13D occur in response to the userpicking up the asset 1317 as illustrated in FIG. 13C. The environment1400 includes the text indicia 1401 that states “look that way,” whichis accompanied by the navigational projection 1403 that illustrates whatdirection the user should walk in in order to sort the asset. FIG. 13Eillustrates the correct location to sort an asset within the environment1400. In some embodiments, the instructions illustrated in FIG. 13Eoccur in response to the user moving responsive to the text indicia 1401and/or the navigational projection 1403 of FIG. 13D. The environment1400 includes the text indicia 1407 “sort here” indicating, along withthe navigation projection 1405, where the correct cell or location forsorting the asset is. FIG. 13F illustrates the combined environments 800and 1400 of FIGS. 13A through 13E. Accordingly, the user 1411 picks upthe asset 1413 responsive to viewing a first set of navigationalprojections and/or text indicia and places the asset 1413 within thecorrect location responsive to viewing a second set of navigationalprojections and/or text indicia.

V. Exemplary System Operation

A. Exemplary User Device Operation

FIGS. 8-13F illustrate an exemplary environment in which assets 10 aremoved amongst various locations 400, which locations may be picklocations (e.g., for storage of an asset in a warehouse or the like;e.g., FIGS. 8-11C in particular), a conveyor belt location (e.g., FIG.12 in particular), and/or sort locations (e.g., for placement of anasset following distribution from a pick location to a conveyor beltlocation; e.g., FIGS. 13A-F in particular). In various embodiments, auser 5 (e.g., sort personnel) may utilize a user device 110 as describedherein while transporting assets 10. As described herein, the userdevice 110 may be configured for receiving information regarding aparticular asset 10 to be transported, whether from the control system100 or otherwise, for guiding the user 5 to a location in which theasset 10 is or should be transported to, and for informing the user 5whether the asset 10 is being located (e.g., via navigationalprojections) and/or transported appropriately.

FIG. 6 illustrates exemplary steps carried out by the user device 110according to various embodiments of the present disclosure to achievethe advantages and capabilities outlined above. In Block 501, aninitialization or calibration of the user device 110 may be conductedaccording to various embodiments. In certain embodiments this step orblock may be optional; in other embodiments, it need only be conductedperiodically, for example upon initial use of the user device and/orupon receipt—from the control system 100—of a notification that anenvironment in which the user device operates has been altered orupdated. As described herein above, per block 501 an environment that auser is located in is mapped based at least in part on generating amultidimensional (e.g., 3-D) graphical representation of theenvironment.

According to various embodiments, with reference to FIG. 6 and FIG. 8 ,during Block 501 of FIG. 6 , the user device 110 may be worn by a user 5so as to map an environment 700 in which the user device 110 is to beused. A graphical representation 701 of the environment may be generatedand/or stored via the user device 110; whereby storing may occur locallyat the user device and/or be stored at and/or synced with the controlsystem 100, for example, upon completion of the mapping procedure. Asmay be understood from FIG. 8 and in some embodiments, the mappingprocedure of step or Block 501 involves the user, while wearing the userdevice 110, to move through the environment 700, which movementnecessarily involves passage of locations 400/1400 (e.g., shelves) andvarious assets or packages 10. During the mapping procedure,three-dimensional depth sensors 119 (e.g., the depth sensors 119 of FIG.5 ) of the user device may be utilized to capture the data required togenerate the graphical representation 701. In conjunction a camera(e.g., the camera 116 of FIG. 5 ), locations of shelving (e.g.,locations 400) and also locations of assets/packages—for example in“pick locations”—may also be established, determined, and/or otherwisesaved at or by the user device 110. Movement of the user 5 through theenvironment 700 defines progressive mapping zones 705, through which thethree dimensional depth sensors are configured to scan during the courseof mapping. To facilitate and further optimize the mapping procedure, incertain embodiments, one or more commands may be transmitted to the user(e.g., via the speaker 117) from the control system (or otherwise), forexample to instruct the user to “turn left” at specific intervalsconfigured to ensure that the entirety of the environment 700 (or adesired portion thereof) is sufficiently mapped. This feature will, forexample, minimize and/or eliminate instances of users not covering everyarea within the environment for which mapping may be desired and/orrequired. In some embodiments, this mapping using 3D scanningcapabilities to detect the immediate environment of the user projectionsare modified to avoid visual distortion. Accordingly, shapes and imagesprojected are adjusted to the shape of the surface they will beprojected upon in particular embodiments. In these embodiments, objectrecognition devices and/or scanners, such as cameras, located within adevice component (e.g., device component 114) can identify the contoursof the environment. In response, a projection can be made in theenvironment based on the shapes (e.g., uneven surfaces) of theenvironment.

Returning to FIG. 6 , in Block (or step) 502, the user device 110 isconfigured to receive and/or associate product (e.g., package or asset)locations within the mapped environment. This may be via utilization ofidentifiers 415 (as described elsewhere herein), via transmission ofasset location information from the control system to the user device,and/or the like.

According to various embodiments, upon completion of the mapping of theenvironment 700 and the association of product (e.g., package or asset10) locations therein, the user device 110 is calibrated for operationalmode or use, which use may occur in either (or both) a pick and a sortmode. In the pick mode, the user device is configured to guide a userthereof to a location in which a package or asset 10 may be picked or“pulled” for fulfillment of an order (e.g., within the environment 800of FIG. 13A); in the sort mode, the user device guides the user (e.g.,from a conveyor belt to a sort location), defined to enable furthertransport and handling of the asset or package in route to a customer orthe like (e.g., within the environment 1400 of FIG. 13D).

If it is determined in block 515 that pick mode is appropriate, the userdevice 110 proceeds to step or Block 503, wherein pick location data isreceived. In certain embodiments, the pick location data is received—atthe user device 110—from the control system 100, for example, uponreceipt—at the control system—of a customer order for a particular asset10 or package. Based upon the received pick location data in Block 503,the user device 110 is configured to, in certain embodiments, generatepick instructions in Block 504. The generation of pick instructions inBlock 504 may entail compilation of a route through which the user ofthe user device 110 must travel—from their present location—so as toreach the location of the asset needing to be picked. Block 504 mayfurther entail generation of a plurality of potential navigationalprojections (e.g., as described in FIGS. 9-11C) that will be necessaryto accurately and efficiently guide or direct the user of the userdevice to the pick location. In Block 504, in certain embodiments,multiple possible routes may be determined and assessed, with either theuser device (automatically) or the user (via an interface selection)choosing an optimal route, for which the navigational projections maythereafter be established. Associated audible commands may also begenerated/established, in conjunction with the navigational projections,should it be desirable to—in addition or alternatively to thenavigational projections—also audibly guide (e.g., via the speaker 117)the user, instructing them to, for example, “turn left after the nextshelving row, as depicted in FIG. 9 , by way of non-limiting example.

Upon completion of step or Block 504 the user device 110 is configuredto proceed to Block 505, wherein the navigational projections and/oraudible instructions are dynamically displayed and/or otherwise providedto the user of the user device, characterized generically as “pickinstructions.” It should be understood that, according to certainembodiments, Blocks 504 and 505 need not be separate and distinct stepsor blocks; instead, as will be described below, as the user movesthrough the environment 700, the user device 110 may be configured todynamically generate and display various navigational projections and/oraudible instructions. In at least those embodiments, Block 504 mayentail merely identifying—at the user device 110—the user's presentlocation, the pick location, and a variety of pathways or routesthere-between.

Reference is made now to FIGS. 9-11C, which depict a variety ofexemplary navigational projections that may be generated and displayedvia the user device 110 as a user 5 wearing the same moves around anenvironment 700. As may be understood by contrasting FIG. 9 with FIGS.10-11C, ahead of the user arriving proximate the pick location, variousnavigational projections 710 may be generated and displayed (e.g., viathe pivoting laser projector 118 of FIG. 5 ). These navigationalprojections 710 may be two or three-dimensional in nature and—asgenerally understood—provide directional guidance to the user, as towhich direction they should move or turn. As illustrated, by way ofnon-limiting example, in FIG. 9 , the navigational projection 710 incertain embodiments may be a two-dimensional directional arrowconfigured to instruct a user waring a user device to make a changerelative to their present movement pattern. In other embodiments, suchas those illustrated in FIGS. 12-13F, the navigational projections,therein described as indicators 810 or navigational projections 810, maybe three-dimensional in nature.

Returning now to FIGS. 9-11C, with reference now in particular to FIGS.10-11C, therein may be seen certain navigational projections 715, 715A,715B, 715C, which may be generated and displayed for the user—via theworn user-device 110—as they approach the proximity of the picklocation. Proximity may be defined as within a specific row upon whichthe asset or package to be picked is located. As may be understood fromFIGS. 11A-11C in particular, the navigational projections 715 mayinclude a frame and/or a checkmark, showing the right location to pickthe asset or package from. In certain embodiments, text indicating aquantity may also be illustrated, as in FIG. 11A via projection 715A,which projection encompasses multiple package or asset boundaries, asmay be recognized and detected by the user device 110 upon approach tothe asset or package. The boundaries may, as a non-limiting example, bedetermined by the user device 110, at least in part, based upon assetinformation received from the control system 100. In other embodimentssoftware embedded upon the user device 110, in conjunction with thecamera (e.g., the camera 116 of FIG. 5 ) may be configured to, via aniterative machine learning-type algorithm and/or object recognitionalgorithm, recognize and determine asset/package dimensions and thusboundaries over a period of time.

FIG. 11A illustrates one exemplary embodiment, in which, as alluded toabove, the navigational projection 715A encompasses multiple assets orpackages 10 in a particular location 400, with textual instructions alsobe generated and provided to the user to “pick 2” of the highlighted orframed packages. In other embodiments (not illustrated), theinstructions may say “pick these two” assets, with the frameencompassing only two specific assets on the shelf or location 400. Instill other embodiments (e.g., FIG. 11C), multiple frames andinstructions associated with navigational projection 715C may beprovided, distinctly identifying three assets or packages that need tobe picked. A single asset pick embodiment is also illustrated in FIG.11B, whereby a navigational projection 715B is provided, similar to theprojection 715C, in that each frame and textual instruction generatedand displayed surrounds and is overlaid relative to a single package orasset 10. It should be understood that the illustrations and embodimentsof FIGS. 11A-C are non-limiting in nature; additional and/oralternatively configured navigational projections 715 may be envisionedwithin the scope of the inventive concept described herein.

Returning now to FIG. 6 and remaining with Block 505, in addition toaudible guidance instructions that may be provided/generated inconjunction with the navigational projections 710, 715 of FIGS. 10-11C,various embodiments may involve the user device 110 further generatingand transmitting to the user 5 audible alerts when deviations occur. Forexample, the user device 110—via its speaker 117 of FIG. 5 —may providegeneral feedback to the user, such as “you picked the wrong asset;please await further instructions” or “you turned the wrong way; pleasestand still pending updated navigational projections becoming visible.”Alternative or additional “feedback” type alerts and/or instructions maybe generated, in a near real time or real time manner, based upon theuser's responsiveness (and accuracy of movement) relative to theprovided navigational projections.

Turning now to step or Block 506 in FIG. 6 , upon the user with the userdevice 110 reaching the pick location, the user device 110 is configuredto further detect asset handling, specifically when the asset or packagehas been picked up by the user. In certain embodiments, feedback mayalso be provided at this junction, via the speaker 117 of the userdevice 110. In these and still other embodiments, the user device 110may capture an image of the “picking” (e.g., via its camera 116) and/ortransmit the same to the control system 100 for centralized/remoteverification of picking accuracy and completeness. This action may occuralso in conjunction with Block 507, whereby one or more pick-relatednotifications may be generated and/or transmitted by the user device110, whether to the user 5 and/or the control system 100.

According to various embodiments, the detection of the “picking” may beconducted by the user device 110 via a collision detection algorithm, asdetailed elsewhere herein. As generally understood, though, suchalgorithm(s) are configured to detect changes in movement relative tothe user device 110, whereby if an item or person (e.g., a user)associated with or wearing the user device 110 encounters acollision—for example by picking up and physically touching an asset orpackage—that “collision” likewise registers at the user device. In thismanner, the user device 100 may be programmed to transition from aguidance mode to—at least temporarily—a report or notification mode, soas to convey—for example to the control system 100—that the pick hasoccurred.

In certain embodiments multiple algorithms may be utilized. One may beto identify what an asset or package is, namely what its physicalboundaries entail. Another is to interpret when a user's hands (or thelike) collide with and pick up (or set down) the asset or package. Eachmay be assisted, not only via the depth sensors 119 of the user device,but also the camera 116 thereof. In certain embodiments, at least twoand in some instances four depth sensors 119 may be provided, each witha working range between 0.85 and 3.1 meters (alternative ranges may alsobe provided). Using data collected via the depth sensors, theenvironment may thus not only be mapped, but changes therein, includingcollisions between objects—including packages, assets, users, and/ordevices such as forklifts operated by a user—may be detected andaccounted for. In some embodiments, particular algorithms able toidentify the parcel itself using machine learning techniques to searchfor physical clues (e.g., size, color, scratches, or any feature, evenmicroscopic that may lead to uniquely id the parcel) without needing toread any parcel id label or barcode.

Remaining with FIG. 6 and returning to Block 515, in certain instancesthe user device 110—whether independently or following a successful pickprocedure—may be configured to operate in a sort mode, which correspondsgenerally to utilization of the user device to move assets or packagesto/from various sorting locations within a mapped environment, ascontrasted with a user locating the asset or package from a stored“pick” location. The sequence in sort mode initiates with the userdevice 110, in Block 508, detecting handling of an asset by the user(e.g., via the collision algorithm described previously herein) and/orobtaining asset identifier data from the asset or package (e.g., via thecamera 116, whether independently at the user device 110 or further inconjunction with an exchange of certain asset/package data with thecontrol system 100).

Upon obtaining the asset identifier data, the user device 110 is able todetermine and/or receive sort location data for the asset or package 10in Block 509. Based thereon, much like in Blocks 504-505 and 507 (in thecontext of sorting), the user device 110 is configured to—according tovarious embodiments—generate sort instructions in Block 510, dynamicallydisplay sort instructions (e.g., navigational projections, text indicia,and/or audible instructions and the like) in Block 511, andgenerate/transmit one or more sort-related notifications in Block 512.In some embodiments, the generating and displaying of one or morenavigational projections configured to guide the user to an assetlocation within an environment is based at least on associating one ormore asset locations within a mapped environment. In variousembodiments, the displaying occurs within an environment that a user isin without regard to a necessary medium (e.g., lens, projector screen,etc.). In these embodiments, the projection is displayed in open spacewithin the environment. It should be understood that any of Blocks509-512 may be substantially the same or identical (the same) as thosein Blocks 503-505 and 507, as previously detailed herein; in certainembodiments, though, one or more of the Blocks may be configureddifferently for sort versus picking mode.

Additional details relative to the utilization of the user device 110 insort mode may be understood with reference to FIGS. 13A-F, which figuresare also described elsewhere herein. Additional details regardingsorting procedures may also be understood with reference to U.S. Ser.No. 15/390,109, the contents of which as are hereby incorporated byreference in their entirety.

B. Exemplary Control System Operation

FIG. 7 illustrates exemplary steps carried out by the control system 100according to various embodiments of the present disclosure. Asillustrated in FIG. 7 , the control system 100 may receive assetidentifier data at Block 601. As indicated herein, the asset indicatordata may be received from the user device 110, the acquisition device810, and/or the one or more location devices 415 at a location 400.Further details regarding the scope and contents of the asset identifierdata have been described previously herein. Still additional details inthis respect may be understood with reference to U.S. Ser. No.15/390,109, the contents of which as are hereby incorporated byreference in their entirety.

At Block 602, the control system 100 may be configured to determine theappropriate location 400 for the asset 10 and/or the appropriateposition within the location for the asset 10. In various embodiments,the determination of the appropriate location for the asset 10 may bebased at least in part on the received asset identifier data. Moreover,the control system 100 may utilize location data corresponding to eachof the locations 400 to determine whether any subsequent processing tobe performed on assets 10 at each location 400 will move the asset 10closer to its final destination. As a non-limiting example, the controlsystem 100 may determine the appropriate location for an asset 10 to bedelivered to 123 Main Street, Atlanta, Ga. is the delivery vehicle thatwill deliver other assets 10 to 123 Main Street, Atlanta, Ga. Additionaldetails in this respect may be understood with reference to U.S. Ser.No. 15/390,109, the contents of which as are hereby incorporated byreference in their entirety.

Referring again to FIG. 7 , at Block 703 the control system 100 may beconfigured to transmit data identifying the appropriate sort location tothe user device 110. As noted herein, the user device 110 may beconfigured to generate an indicator (e.g., visual indicators ornavigational projections 710/715/810) discernible by the user 5 (e.g.,carrier personnel) regarding the appropriate pick or sort location forthe asset 10. However, as noted herein, each asset 10 may haveinformation indicative of an appropriate location printed thereon, andaccordingly the control system 100 may not need to—in thoseembodiments—transmit appropriate location data to the user device 110.

The control system 100 may also be configured to receive a variety ofdata—including location data—from the user device 110 at Block 604. AtBlock 605, the control system 100 may subsequently compare theappropriate location (at which the user for picking or the asset forsorting should be located) and the actual location data received atBlock 604 to determine whether the user device 110 is proximate theappropriate location. As indicated at Block 606, the remaining steps tobe completed may be selected based at least in part on a determinationof whether the location is an appropriate (or desired/accurate)location. Additional details in this respect may be understood withreference to U.S. Ser. No. 15/390,109, the contents of which as arehereby incorporated by reference in their entirety.

Upon a determination that the user device 110 is proximate an incorrectlocation 400, the control system 100 may generate mistake data at Block610. Upon generating the mistake data, the control system 100 maytransmit the mistake data to the user device 110 at Block 611. Asindicated herein, the user device 110 may be configured to generate amessage discernible by the user 5 (e.g., carrier personnel) indicatingthe user device 110 is proximate an incorrect location 400 (e.g., asillustrated in FIG. 13D). In various embodiments, the control system 100may be configured to associate the asset identifier data with thelocation data corresponding to the location 400 at Block 612. At Block613, the user 5 may continue transporting the asset 10 (and consequentlythe user device 110) to another (ideally correct) location 400. Theprocess may return to Block 601 in such scenarios and repeat the recitedsteps.

Referring again to Block 606, the process may proceed after comparingthe actual/received location data and the appropriate location data forthe asset 10 (illustrated as Block 605) with reference to Blocks 607-609if the user 5 approaches the appropriate location. In the context ofsorting procedures, additional details in this respect may be understoodwith reference to U.S. Ser. No. 15/390,109, the contents of which as arehereby incorporated by reference in their entirety.

The control system 100 may be further configured to generate one or morealerts regarding the association between the asset identifier data andthe location data. The control system 100 may be configured to generatean alert to inform the user 5 (e.g., carrier personnel) or other usersregarding asset identifier data being associated with location data.

VI. Conclusion

Many modifications and other embodiments of the disclosures set forthherein will come to mind to one skilled in the art to which thesedisclosures pertain having the benefit of the teachings presented in theforegoing descriptions and the associated drawings. Therefore, it is tobe understood that the disclosures are not to be limited to the specificembodiments disclosed and that modifications and other embodiments areintended to be included within the scope of the appended claims.Although specific terms are employed herein, they are used in a genericand descriptive sense only and not for purposes of limitation.

What is claimed is:
 1. A system for providing hands-free handling of atleast one asset by a user, the system comprising: a user deviceconfigured to be worn by a user, the user device comprising: one or morememories; and one or more processors configured to: obtain assetidentifier data for the at least one asset, the user device and the atleast one asset being located within a physical environment; based atleast in part on the obtained asset identifier data, determine locationdata associated with a location for the at least one asset, the locationbeing within the physical environment; based at least in part on thelocation data, generate and display, over a surface within the physicalenvironment, one or more navigational projections, the one or morenavigational projections being configured to guide the user to thelocation; subsequent to the generation and display of the one or morenavigational projections, detect handling of the at least one asset bythe user; and receive one or more notifications associated with thehandling of the at least one asset by the user.
 2. The system of claim1, wherein the user device further comprises a mounting mechanismconfigured for mounting the user device to either a chest or a topportion of a head of the user.
 3. The system of claim 1, wherein theuser device further comprises: a plurality of depth sensors; and apivoting laser projector, wherein: the pivoting laser projector is usedfor the displaying of the one or more navigational projections; and theone or more navigational projections are made based on the location dataof the user device collected via the plurality of depth sensors.
 4. Thesystem of claim 1, wherein the one or more navigational projections aretwo-dimensional projections generated via a pivoting laser projector ofthe user device.
 5. The system of claim 1, wherein the one or morenavigational projections are three-dimensional projections generated viaa pivoting laser projector of the user device.
 6. The system of claim 1,wherein the one or more navigational projections are holographic imagesgenerated via a pivoting laser projector of the user device.
 7. Thesystem of claim 1, wherein the user device further comprises a pluralityof depth sensors configured to map at least a portion of the physicalenvironment to calibrate, prior to utilization of the user device, theuser device in an operational mode for asset picking or sorting.
 8. Thesystem of claim 1, wherein the user device further comprises at leastone speaker configured to cause a transmission of audible instructionsto the user concurrent with the displaying of the one or morenavigational projections.
 9. The system of claim 8, wherein the audibleinstructions are configured to provide feedback in response todetection, by the user device, of a deviation of the user from the oneor more navigational projections.
 10. A computer-implemented methodcomprising: obtaining asset identifier data for at least one asset, theat least one asset being located within a physical environment;determining location data associated with a location for the at leastone asset, the location being within the physical environment; based atleast in part on the location data, causing display, over a surfacewithin the physical environment, of one or more navigationalprojections, the one or more navigational projections being configuredto guide a user to the location; subsequent to the causing display ofthe one or more navigational projections, detecting handling of the atleast one asset; and determining one or more notifications associatedwith the handling of the at least one asset.
 11. Thecomputer-implemented method of claim 10, wherein a pivoting laserprojector is used for the causing displaying of the one or morenavigational projections.
 12. The computer-implemented method of claim10, wherein a control system is used for the causing display of the oneor more navigation projections.
 13. The computer-implemented method ofclaim 10, wherein the one or more navigational projections aretwo-dimensional projections generated via a pivoting laser projector ofa user device.
 14. The computer-implemented method of claim 10, whereinthe one or more navigational projections are three-dimensionalprojections generated via a pivoting laser projector of a user device.15. The computer-implemented method of claim 10, wherein the one or morenavigational projections are holographic images.
 16. Thecomputer-implemented method of claim 10, further comprising mapping atleast a portion of the physical environment to calibrate a user devicein an operational mode for asset picking or sorting.
 17. Thecomputer-implemented method of claim 10, further comprising causing atransmission of audible instructions to the user concurrent with thecausing display of the one or more navigational projections.
 18. Thecomputer-implemented method of claim 17, wherein the audibleinstructions are configured to provide feedback in response to detectionof a deviation of the user from the one or more navigationalprojections.
 19. A computer program product for providing hands-freehandling of at least one asset by a user, the computer program productcomprising at least one non-transitory computer-readable storage mediumhaving computer-readable program code portions stored therein, thecomputer-readable program code portions comprising one or moreexecutable portions configured for: obtaining asset identifier data forthe at least one asset, the at least one asset being located within aphysical environment; based at least in part on the obtained assetidentifier data, determining location data associated with a locationfor the at least one asset, the location being within the physicalenvironment; based at least in part on the location data, causingdisplay, over a surface within the physical environment, one or morenavigational projections, the one or more navigational projections beingconfigured to guide the user to the location; detecting handling of theat least one asset; and determining one or more notifications associatedwith the handling of the at least one asset.
 20. The computer programproduct of claim 19, wherein the computer program product is included ina user device, the user device comprises a mounting mechanism configuredfor mounting the user device to either a chest or a top portion of ahead of the user.